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Abstract 

Command  and  Control  (C2)  organizations  must  operate  decisively  and  synchronously,  and  do  so 
in  highly  uncertain  and  dynamic  settings.  Individuals  succeed  in  these  settings  in  part  by  thinking 
critically  about  their  assessments  and  plans.  We  argue  that  individual  team  members  collaborate 
in  their  application  of  critical  thinking  in  a  process  called  “collaborative  critical  thinking.”  We 
are  developing  this  concept  of  Collaborative  Critical  Thinking  (CCT)  within  C2  teams  from 
three  research  threads  concerning  1)  individual  critical  thinking,  2)  team  process  and 
architecture,  and  3)  human  performance  in  information  age  warfare.  This  paper  describes  our 
CCT  framework  and  the  tools  and  training  we  are  developing  to  improve  CCT  among  team 
members. 


Introduction 

A  fundamental  goal  of  the  military  is  to  ensure  that  Command  and  Control  (C2)  organizations 
operate  decisively  and  synchronously  in  highly  uncertain  and  dynamic  settings.  Individuals 
succeed  in  these  settings  by  thinking  critically;  that  is  by  critiquing  their  understanding  of  the 
situation  at  hand,  refining  their  knowledge,  and  adapting  their  decision  making  and  planning  to 
the  problems  at  hand.  This  project  purports  that  individual  team  members  collaborate  in  their 
application  of  critical  thinking  in  a  process  called  “collaborative  critical  thinking.” 

There  is,  to  our  knowledge,  no  theoretical  framework  to  help  warfighters  learn,  measure,  and 
manage  collaborative  critical  thinking  (CCT).  However,  aspects  of  this  topic  are  addressed  by 
recent  programs  of  research  concerning  critical  thinking,  team  processes  and  architecture,  human 
performance  in  infonnation  warfare. 

A  theory  about  how  individual  warfighters  make  decisions  under  uncertainty  has  been  validated 
through  research  concerning  critical  thinking.  This  recognition-metacognition  framework 
(Cohen,  Freeman,  et  ah,  1996,  1997,  1998)  speculates  that  expert  warfighters  monitor  for 
opportunities  to  critique  their  assessments  and  plans,  identify  sources  of  uncertainty  (i.e.,  gaps, 
untested  assumptions,  and  conflicting  interpretations),  and  reduce  or  shift  that  uncertainty  by 
doing  such  things  as  gathering  information,  testing  assumptions,  and  fonning  contingency  plans 
before  taking  action. 

Research  conducted  under  ONR’s  A2C2  program  and  other  Air  Force  Research  Laboratories 
human  engineering  projects  concerning  team  process  and  architecture  have  produced  a  rich 
body  of  measures  concerning  the  processes  by  which  teams  coordinate  their  activities.  This  team 
coordination  may  occur  either  explicitly,  that  is,  through  explicit  communication,  or  implicitly, 
through  reliance  on  shared  infonnation,  shared  interpretations  of  information  patterns,  and 
standardized  responses  to  those  patterns. 

Recent  theory  and  fieldwork  studying  individual  and  collaborative  cognition  in  command  and 
control  and  other  infonnation-intensive  organizations  has  provided  insights  into  human 
performance  in  information  warfare  by  studying  the  processes  by  which  team  members  may 
interpret  data  to  develop  information,  build  understanding  that  informs  decisions,  and 
collaborate  to  ensure  that  information  and  knowledge  are  shared  in  support  of  synchronized 
action  to  shape  events.  Alberts,  Garstka,  Hayes,  and  Signori  (2001)  have  developed  a  framework 
that  clearly  defines  these  (italicized)  constructs  as  primitives  of  perfonnance  in  information  age 
warfare. 


In  a  research  and  development  project  for  the  Office  of  Naval  Research,  we  are  weaving  these 
threads  together  to  create  a  theory,  validated  measures,  tools  and  training  that  will  help  us  to 
understand  and  support  team  critical  thinking. 

This  paper  addresses  the  theory  development  as  well  as  initial  ideas  about  the  tools  that  will  be 
developed  based  on  this  theory. 


Collaboration  Framework 

Collaboration  is  the  process  of  shared  creation  (Rawlings,  2000).  During  collaboration,  people 
with  complementary  skills  interact  to  create  a  shared  understanding  of  the  situation  that  is  more 
complete  than  would  have  been  achieved  with  any  individual  alone.  The  framework  in  Figure  1 
was  created  in  an  attempt  to  better  understand  the  entire  collaborative  process. 

As  this  framework  suggests,  collaboration  is  not  always  needed;  in  fact,  when  all  the  important 
information  resides  in  one  person  and  time  is  short,  the  collaborative  process  is  less  optimal  than 
a  quick,  autocratic  decision  (Vroom  &  Jago,  1988).  If  expertise  is  distributed  among  multiple 
people,  and  resources  and  responsibilities  are  divided  between  people  as  well,  collaboration  is 
needed.  These  “factors  generating  the  need  to  collaborate”  affect  not  only  the  importance  of 
collaboration,  but  also  the  goals  of  collaboration,  which  will  vary  with  the  responsibilities 
allocated  to  collaborating  parties. 


Figure  1.  Collaboration  Framework 

A  team’s  ability  to  achieve  its  collaboration  goals  depends  on  a  number  of  factors  such  as 
available  technology,  team  members’  collaboration  skills,  and  team  composition  factors. 

Technology.  The  distribution  of  teams  in  time  and/or  space  requires  the  use  of  technology  to 
enable  the  information  sharing  necessary  for  effective  collaboration.  Such  “technology” 
currently  ranges  from  telephones  and  email  to  video-conferencing.  New  technologies  are  being 
developed  that  claim  to  assist  team  collaboration  and  information  sharing  even  more. 


Process/Skill  Factors.  Although  really  effective  teams  can  make  collaboration  look  easy, 
collaboration  is  not  innate.  Effective  collaboration  requires  specific  skills  such  as  negotiation 
and  active  listening. 

Team  Composition  Factors.  Collaboration  is  strongly  affected  by  the  composition  of  the  team. 
For  example,  teams  composed  of  members  with  homogenous  backgrounds  find  early 
collaboration  to  be  much  easier  than  heterogeneous  teams;  however,  heterogeneous  teams 
eventually  make  decisions  that  are  of  a  higher  quality  than  those  of  homogenous  teams. 

These  “factors  affecting  the  ability  to  collaborate”  directly  affect  the  quality  of  the  collaboration 
products  (e.g.,  the  assessments  and  plans)  as  well  as  the  effects  (Measures  of  Effectiveness,  or 
MOEs)  of  team  performance. 

Collaborative  critical  thinking  can  be  thought  of  as  one  factor  that  affects  the  team’s  ability  to 
collaborate  (see  navy  boxes  in  Figure  1).  The  next  section  of  this  paper  describes  this  in  more 
detail. 


Collaborative  Critical  Thinking 

Cohen  developed  a  framework  for  understanding  critical  thinking  that  he  and  colleagues 
validated  in  studies  of  individuals  and  teams,  largely  in  air  combat  (Cohen,  Freeman,  and 
Thompson,  1998)  and  commercial  air  settings  (Freeman,  Cohen,  and  Thompson,  1998).  This 
model  posits  that  critical  thinking  monitors,  tests,  and  refines  recognitional  decision  making  (the 
phenomenon  described  in  the  literature  on  “naturalistic  decision  making”  (Klein,  1993)).  This 
framework  specifies  several  cognitive  functions  that  constitute  critical  thinking;  monitoring, 
critiquing,  and  actions.  Monitoring  tests  for  the  need  and  opportunity  to  critique  assessments  and 
plans.  When  the  decision  maker  senses  uncertainty  and  the  stakes  are  high,  there  is  a  need  to 
engage  in  critical  thinking;  when  time  is  available  to  do  so,  there  is  an  opportunity.  Critiquing 
ferrets  out  specific  sources  of  uncertainty,  such  as  gaps  in  knowledge,  untested  assumptions,  and 
the  existence  of  conflicting  interpretations  of  events.  Actions  such  as  gathering  information  and 
formulating  contingency  plans  manage  -  and  may  even  reduce  -  uncertainty  and  attendant  risks. 

Individuals  engage  in  critical  thinking  within  team  settings.  This  is  well  established  (Freeman, 
Cohen,  and  Thompson,  1998).  We  hypothesize  that  analogs  of  these  critical  thinking  functions 
exist  at  the  team  level,  as  well.  That  is,  high  functioning  teams  encourage  productive,  timely, 
critical  dialogue  between  team  members.  As  markers  of  collaborative  critical  thinking,  we  look 
for  the  following  behaviors: 

Team  leaders  and  members  use  disagreement  within  the  group  as  an  indicator  that  conflicting 
interpretations  should  be  resolved,  that  the  relevancy  and  accuracy  of  assumptions  and 
paradigms  should  be  tested,  and  that  uniquely  held  information  should  be  shared.  Disagreements 
are  pointers  to  uncertainty  in  the  assessments  and  plans  the  team  generates.  Proficient  teams  use 
disagreement  as  an  indicator  when  they  monitor  the  need  to  engage  in  collaborative  critical 
thought. 

Team  leaders  and  members  prioritize  activities  to  ensure  that  time  and  efforts  are  invested  in 
addressing  the  most  critical  sources  of  uncertainty  and  risk.  Thus,  they  monitor  the  opportunities 
to  engage  in  collaborative  critical  thinking,  and  attempt  to  control  them. 

Team  leaders  and  members  engage  in  dialogues  to  identify  the  sources  of  uncertainty  that  might 
weaken  their  assessments  and  plans,  and  they  devise  ways  to  reduce  that  uncertainty  (e.g.,  by 


gathering  information)  or  compensate  for  it  (e.g.,  by  developing  contingency  plans).  In  this  way, 
proficient  teams  critique  their  work. 

Proficient  teams  act  in  coordination  to  gather  and  disseminate  information  as  they  manage 
uncertainty. 

To  document  these  hypotheses,  we  are  conducting  empirical  research  with  Michael  Coovert  and 
his  colleagues  at  the  University  of  Central  Florida.  To  support  CCT  -  particularly  the  functions 
above  -  we  are  developing  tools  and  training.  Both  are  designed  to  help  distributed  team 
members  critique  their  assessments  and  plans.  Both  employ  questions  designed  to  support 
monitoring,  so  that  teams  can  spot  opportunities  to  engage  in  collaborative  critical  thinking  (and 
avoid  needless  debates). 


CCT  Tool 

The  tool  being  developed  will  help  teams  monitor  for  opportunities  to  engage  in  CCT,  support 
CCT  when  it  occurs,  and  thus  improve  the  products  of  team  collaboration. 

The  tool  being  developed  has  two  major  users:  the  team  members  and  a  person  acting  as 
facilitator.  The  team  members  are  asked  questions  at  crucial  points  during  the  collaboration 
process  that  give  insight  into  the  current  state  of  CCT  (Figure  2  shows  a  prototype).  Answers  are 
made  anonymously,  to  encourage  candor  -  a  precious  but  scarce  commodity  in  some  settings. 
Each  question  consists  of  a  rated  item  and  an  optional  comment.  The  system  analyzes  the  rated 
items  and  distributes  descriptive  analyses  plus  comments  to  the  facilitator,  who  then  decides 
whether  action  should  be  taken  (Figure  3).  Alternatively,  they  may  be  distributed  to  the  team  to 
foster  awareness  of  the  team’s  state. 


Figure  2.  Screenshot  of  initial  team  member  interface 


Figure  3.  Screenshot  of  facilitator’s  interface 

What  questions  might  this  system  present  to  users  that  would  support  collaborative  critical 
thinking?  Consider  a  decision,  made  during  planning  of  an  air  campaign,  whether  to  gather  battle 
damage  assessments  using  special  forces  on  the  ground  (a  tactic  that  produces  highly  accurate 
reports  at  high  risk  to  warfighters)  or  by  other  means,  such  Predator  UAVs  (an  approach  that  can 
produce  less  accurate  reports  in  some  weather  conditions  but  does  not  expose  human  warfighters 
to  threats).  While  the  team  researches  the  targets  and  enemy  defenses,  and  discusses  these 
options,  the  team  leader  polls  the  members,  asking,  “Are  you  confident  in  the  plan  to  use  ground 
observers?”  Respondents  answer  this  question  by  selecting  a  rating  on  a  Likert  scale  anchored  on 
one  end  by  “No”  and  the  other  by  “Yes.”  Low  variance  in  responses  to  this  question  (whether  the 
ratings  are  high,  low,  or  at  the  mean)  suggest  that  there  is  little  uncertainty  to  be  resolved  on  this 
question.  High  variance  indicates,  of  course,  that  there  are  differences  within  the  team  that  may, 
if  discussed,  expose  potential  problems  in  the  use  of  ground  observers,  relative  to  other  solutions. 

We  can  also  employ  a  second  order  question  in  this  circumstance.  “Is  there  consensus  within  the 
team  concerning  the  use  of  ground  observers?”  Ratings  gathered  from  team  leaders  in  response 
to  this  question  give  insight  into  their  awareness  of  the  team  state  when  considered  in  light  of 
team  member  responses  to  the  question  above.  Leaders’  awareness  of  team  state  is  accurate 
when  estimates  of  consensus  are  low  and  team  member  confidence  varies  widely,  or  when 
leaders’  estimates  of  consensus  are  high  and  confidence  exhibits  little  variance.  Team  leaders 
may  waste  the  time  and  tax  the  patience  of  team  members  in  unneeded  discussions  or  research 
when  they  estimate  that  consensus  is  low  though  variance  in  confidence  is  low.  Conversely, 


leaders  may  fail  to  identify  and  address  critical  problems  when  they  estimate  that  consensus  is 
high  when  confidence  varies  widely. 

In  time-critical  missions,  it  may  be  useful  to  encourage  sensitivity  to  the  time  available  for 
decision  making  (and,  thus,  for  collaborative  critical  thinking)  and  to  elicit  estimates  of  available 
decision  time  from  team  members  with  different  perspectives  on  the  problem  or  different  levels 
of  expertise.  Uniformly  high  estimates  of  available  time  indicate  that  there  may  be  opportunities 
to  critique  assessments  and  plans.  High  variance  among  estimates  indicate  either  that  some  team 
members  have  time  constraints  not  understood  by  their  team  members,  or  that  some  team 
individuals  misperceive  the  time  course  of  the  mission.  These  are  not  rare  or  trivial  failures.  To 
avoid  them,  experts  in  anti-air  warfare,  for  example,  maintain  an  awareness  of  the  available 
decision  time  so  that  they  can  use  it  well  when  the  stakes  are  high,  and  even  buy  time  when 
possible  (Cohen,  Freeman,  and  Thompson,  1998).  This  sensitivity  to  decision  time  varies  with 
expertise.  An  experimental  test  of  decision  making  by  airline  pilots  (Freeman,  Cohen,  and 
Thompson,  1998)  revealed  that  there  is  marked  difference  in  the  sensitivity  to  time  (but  not 
accuracy  of  time  estimates)  as  a  function  of  experience.  Pilots  with  more  than  20  hears 
experience  used  all  of  the  available  decision  time  (rarefy  much  less  or  much  more)  to  gather 
information  and  make  a  decision  whether  to  divert  to  an  alternate  airport;  pilots  with  less 
experienced  were  more  likely  to  make  hasty  decisions  when  time  was  available,  and  tardy 
decisions  when  it  was  not.  Evidence  such  as  this  suggests  that  teams  may  benefit  from  tools  or 
training  that  help  them  to  monitor  the  available  decision  time  and  use  it  well. 

To  complement  questions  such  as  these,  which  support  monitoring,  we  are  designing  questions 
that  help  teams  to  ferret  out  problems,  identify  actions  to  resolve  problems  or  compensate  for 
them,  coordinate  those  actions,  and  apply  what  they  leam  to  improve  their  assessments  and 
plans.  The  system  will  help  leaders  to  develop  these  questions  by  presenting  question  templates 
that  leaders  specify  to  the  problem  at  hand. 

To  guarantee  that  the  users  get  the  optimal  benefit  from  the  CCT  tool,  training  is  being 
developed  that  will  help  the  users  leverage  these  tools.  An  effective  training  program,  such  as 
the  successful  Team  Adaptation  and  Coordination  Training  (TACT)  Program  (Entin  and  Serfaty, 
1999),  includes  four  components:  lecture,  demonstration,  practice,  and  feedback.  We  plan  to 
build  on  our  aviation  experience  with  cockpit  resource  management  (i.e.,  narrative-based 
realistic  operational  scenarios)  to  create  stories  that  illustrate  critical  elements  necessary  in  CCT. 
We  will  incorporate  these  stories  into  a  training  program  that  will  build  the  necessary  CCT  skills 
and  facilitate  the  tool  integration.  The  validation  planned  for  this  intervention  should  show  that 
teams  with  both  CCT  training  and  the  CCT  tool  achieve  a  better  shared  situation  awareness, 
produce  more  robust  plans,  and  execute  missions  with  greater  synchrony  and  success  than  teams 
that  do  not  receive  these  CCT  interventions. 

References 

Alberts,  David  S.,  Garstka,  J.J.,  Hayes,  R.E.,  and  Signori,  D.A.  (2001).  Understanding 
Information  Age  Warfare,  www.dodccrp.org. 

Cohen,  M.S.  &  Freeman,  J.T.  (1997).  Improving  Critical  Thinking.  In  Flin,  R.,  et  al.  (eds.), 
Decision  Making  Under  Stress:  Emerging  Themes  and  Applications.  Brookfield,  VT:  Ashgate 
Publishing  Co. 


Cohen,  M.S.,  Freeman,  J.T.  and  Thompson,  B.T.  (1998).  Critical  Thinking  Skills  in  Tactical 
Decision  Making:  A  Model  and  A  Training  Method.  (Canon-Bowers,  J.  and  E.  Salas,  eds.), 
Decision-Making  Under  Stress:  Implications  for  Training  &  Simulation.  Washington,  DC: 
American  Psychological  Association  Publications. 

Cohen,  Marvin  S.,  Freeman,  Jared  T.  and  Wolf,  Steve.  (1996).  Meta-recognition  in  time- 
stressed  decision  making:  Recognizing,  critiquing,  and  correcting.  Journal  of  the  Human 
Factors  and  Ergonomics  Society. 

Entin,  E.  E.  &  Serfaty,  D.  (1999)  Adaptative  Team  Coordination  Human  Factors,  41,  2.  312- 
325. 

Freeman,  J.  Thompson,  B.,  Littleton,  E.B.,  Craig,  P.,  Rubineau,  B.,  Bailin,  S.,  Serfaty,  D.,  and 
Cohen,  M.S.  (2000).  Metrics  for  Evaluation  of  Cognitive  Architecture-Based  Collaboration 
Tools.  Aptima  Technical  Report  AP-R-1 1 19.  Wobum,  MA. 

Freeman,  J.,  Cohen,  M.S.,  and  Thompson,  B.T.  (1998).  Effects  of  Decision  Support  Technology 
and  Training  on  Tactical  Decision  Making.  Proceedings  of  the  1998  Command  and  Control 
Research  &  Technology  Symposium,  Monterey,  CA 

Freeman,  J.,  Cohen,  M.S.,  and  Thompson,  B.T.  (1998).  Time-Stressed  Decision-Making  in  the 
Cockpit.  Proceedings  of  the  Association  for  Information  Systems  1998  Americas  Conference, 
Baltimore,  MD. 

Klein,  G.A.  (1993),  ‘A  Recognition-Primed  Decision  (RPD)  model  of  rapid  decision  making.’  In 
G.A.  Klein,  J.  Orasanu,  R.  Calderwood  &  C.E.  Zsambok  (Eds.),  Decision  Making  in  Action: 
Models  and  Methods,  Norwood,  NJ:  Ablex  Publishing  Corporation.,  pp.  138-147. 

Rawlings,  D.  (2000).  Collaborative  leadership  teams:  Oxymoron  or  new  paradigm?  Consulting 
Psychology  Journal,  52,  36-48. 

Vroom,  V.  H.  and  Jago,  A.  G.  (1988).  On  the  validity  of  the  Vroom-Yetton  model.  Journal  of 
Applied  Psychology,  63,  151-162. 


Collaborative  Critical  Thinking 

8th  International  Command  and  Control 
Research  and  Technology  Symposium 

June  17  -  19,  2003 
National  Defense  University 
Washington,  DC 

Jared  Freeman,  Ph.D., Kathleen  P.  Hess,  Ph.D., 
Gabriel  Spitz,  Sc.D.,  Michael  J.  Garrity, 
Gavan  Lintern,  Ph.D.  :  Aptima,  Inc. 

Michael  D.  Coovert,  Ph.D., Thomas  G.  Gordon,  Ph.D., 
Jason  Read,  Dawn  Riddle,  Tim  Willis,  Thomas  S. 
King  V.:  University  of  South  Florida 


Overview 
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■  A  Collaborative  Critical  Thinking 
framework  for 

■  understanding 

■  measuring 

■  training  and 

■  supporting 

■  Technology 

■  Experimentation 
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Collaborative  Critical  Thinking  Framework 


Foundations  in  Science  &  Theory 


Information  age  warfare 

■  Teams  are  distributed,  ad  hoc,  multi¬ 
disciplinary,  mission-critical 

■  Teams  require  coordination  &  collaboration 

■  Manage  forces  &  information 

■  Achieve  effects 

■  Supporting  coordination  &  collaboration 
requires  measurement 

Coordination  &  collaboration  processes  can 
be  measured 

Collaboration  often  involves  critical 
thinking.  For  individuals,  CT 

■  Is  found  in  transcripts  of  planning 

■  Can  be  trained 

■  Improves  mission  performance  in  Air 
Defense  scenarios 


■  Alberts,  Garstka, 
Hayes,  and 
Signori  (2001) 

Letsky  et  al. 
(2003) 


Macmillan,  et  al., 
2001 

■  Miller,  Price, 
Entin,  & 
Rubineau,  2001 

Moon,  et  al., 
2000 


Cohen, 

Freeman,  and 
Thompson,  1998 


Cohen  and  < 
Freeman,  1997 
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■  Collaboration 

■  Consists  of  functions  (or 
processes) ... 

■  That  effect  C2  . . . 

■  Which  produces  mission 
effects 

■  What  are  collaboration 
functions,  particularly 
Collaborative  Critical 
Thinking? 


Overview:  Collaboration 
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■  Collaboration 
involves* 

1.  Process  knowledge 

2.  Domain  knowledge 

3.  Team  knowledge 

4.  Negotiating 
solutions 

5.  Testing  &  revising 
solutions 


*(Letsky  et  al.,  2002) 


Framework  1:  Collaboration 
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Framework  2:  Collaborative  Critical  Thinking 

Collaborative  critical 
thinking*  engages  multiple 
team  members  in 

■  Monitoring  for  uncertainty 

■  Detecting  opportunities  to 
handle  it 

■  Specifying  problems 

■  Solving  problems  &  gathering 
info 

CCT  can  be  applied  to 

■  Assessments 

■  Plans 

■  The  team  process  &  structure 


*Freeman,  et  al.,  2001,  2002;  Cohen, 
et  al.  1997,  1998 
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Framework  3:  Dispositions  Support  Critical  Thinking 


Critical  thinking  skills  may  be 
driven  (in  part)  by 
dispositions* 

■  systematic  inquisition  to  find 
truth 

Measures 

■  Observational 

■  Standardized  instruments 

■  Self  report 


*  Facione,  1998 


Collaborative  Critical  Thinking 
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■  Define,  Measure, 

Train  and  Support 
Collaborative  Critical 
Thinking 

■  Measure  its  effects  on 
C2  &  Mission 
outcomes 


Aptii 

A  Huaan 


11a 

Himin  •  C«M«r*d  E*t<nt«rin« 


CCT  Decision  Support 


Concept  for  a  CCT  Support  Tool 
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■  The  setting 

■  A  geographically  distributed  team  in  a  long  working 
session 

■  The  team  leader  wants  to  monitor  CCT  activity 

■  Team  members  need  reminders  to  engage  in  CCT 

■  Two  components 

■  Respondents'  tool  - 

■  Elicits  data  concerning  team  member  monitoring, 
assessments,  critiques,  actions 

■  Cues  team  members  to  monitor,  assess,  critique,  act 

■  Leader's  tool 

■  Helps  leader  or  aid  plan,  poll  for,  and  analyze  collaborative 
critical  thinking  activity 
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Workspaces  in  a  Distributed  Team 
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A  Pop-Up  Probe 


*i  Mission  Planning  Collaborative  Tool 


File  Edit  Window  Help 


J^jxJ 


G> O 


_ ail 

[This  is  the  question  to  the 
participant 

_  III . I  I 

Comment  |  OK 
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Opportunity  to  Rate  and  Comment 


is  CCL  -  Evaluate 


Mission  Planning  Collaborative  Tool 

^nj*l| 

File  Edit  Window  Help 

B\  *| 

|Aj 

LltJ 

121 

IP 

l&j 

||  This  is  the  context  question  for  the  comment 

Comments: 


Canned 


•  °J 

^67 


$><$  @0  05:23  PM 
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«j  Collaborator  -  Monitor  —  Objective:  Develop  Effective  COA  to  Deal  with  Emerging  SAM  Site 


File  Edit  Tools  Window  Help 


Rating  Results  +  Advice 

-Ini  x| 


Current  Status 


Opinion: 


This  is  where  the  question 
will  go 

J 

i 

J  | 

Comments: 

Thte  areX  comments 


1 )  Text  of  the  first  comment 


Polling: 


ii  Collaborator  -  Advice 


*J 


Next  poll  ir 
Next  quest 


This  is  whe 
next  questi 
displayed 


Overall  Status 


Current  time:  1 0930 


Opinion  Data 

Data  view:  [Participants  by  Attributes 


Team  &  Leader  Status  re:  Plan  A 


Percewed  uncertainty 
1- 


Action  plan  i — — S— f - K^nTime  available 


Sources  of  uncertainty 


- TeamLow 

- Team  High 

- Leaders):  Low 

- Leaders):  High 


Advice ... 


Refresh 


Save  snapshot 


To... 


Team  Leader 


Some  team  members  believe  the  plan  is  flawed.  Time  is  available  to 
critique  the  plan.  Use  it 


Cancel 


Send 
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A  Summary  of  Comments 
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Certified  fnjirtferinj 


Participation  Statistics 
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Leader's  Configuration  Interfaces 


•a  Collaborator  -  Critical  Thinking  Assessment  Set  Up 


Work  Session  |  Probes  ;  Participants;! 


X] 


Participants  in  the  critical  thinking  assessment: 


I-  Anastasi  Donna 

r  Bailey  Adam 

f~  Baker  Keith 

P  Chopra  Kari 

f  Hight  Heather 

LevchukYuti 

|“  Miller  Diane 

“  Not  all  participants  in  the  work  session  need  to  participate  in  the 
critical  thinking  assessment 


Coordinator  of  critical 
thinking  assessment: 

Leader  of  work  session: 


Configure  (clockwise) 

■  Work  session 

■  Probes  &  schedule 

■  Participants 
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Huas*  Ctffltrtd  E«0«*ttnnf 


CCT  Probes 


Criteria  for  CCT  Probes 
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■  Probes  consist  of 

■  CCT  template:  "Are  you  confident  in  the 
plan..." 

■  Mission-specific  content:  "to  use  ground 
observers  to  assess  battle  damage ?" 

■  Probes  measure  CCT  with  quick  ratings  re: 

■  Monitoring,  Assessing,  Critiquing,  Action 

■  Probes  elicit  CCT 

■  Comments 

■  Team  CCT  states  ->  diagnosis  &  action 


Monitoring  for  Uncertainty 


Aptima 

^  Hunan  ■  Cantatas  Engmatrmf 


Team:  "Are  you  confident  in  the  plan  to  use 
ground  observers  to  assess  battle  damage ?" 
(0=No  10=Yes) 

■  Low  variance  =  Consensus.  Don't  invest  time  in 
critiquing  this  plan  unless  the  situation 
changes. 

■  High  variance  =  Little  agreement,  high 
uncertainty.  If  time  is  available  to  critique  the 
plan,  focus  here. 


■  Analysis  &  Advice 

■  High  confidence,  strong  consensus 

-  Advice:  Do  not  invest  time  in  critiques  on  this 
issue  unless  the  situation  changes  significantly. 

■  Low  confidence,  strong  consensus 

-  Advice:  Critique  this  issue  if  time  allows. 

■  Weak  consensus 

■  Advice:  Poll  team  members  with  low  and  high 
confidence  to  identify  misperceptions  or 


problems 

0 

Range  of  team  ratings  of 
confidence  in  plan 


Confidence 

Low  High 


Assessing  Available  Time 


Aptiim 

A  Huai  A  ■  Ciniiita  Engtnurini 


■  "How  much  time  is  available  before 
the  team  must  commit  to  a  decision 
concerning  use  of  ground  observers 
for  BDA1" 

•  Low  estimates,  low  variance 

•  No  time  to  critique  plans.  Don't  do  so. 


Time  available 
None  6  hrs 


■  High  variance 

-  Advice:  Some  team  members  have  time 
constraints  not  understood  by  their  team 
members.  Have  team  members  discuss 
their  time  constraints. 

■  Advice:  Some  team  individuals 
misperceive  the  time  course  of  the 
mission.  Talk  with  the  team  members 
with  the  tightest  time  constraints  to 
understand  if  their  constraints  are 
realistic.  Correct  this. 
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Experiment 


Experiment 
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■  Objective: 

■  Determine  the  relative  importance  of  cognitive  and 
dispositional  factors  in  CCT. 

■  Determine  the  impact  on  C2  and  mission  outcomes  of 

■  Training  cognitive  factors  and 

■  Sensitizing  dispositional  factors 

■  Method: 

■  Each  of  3  team  members  receive 

■  Training  in  several  cognitive  aspects, 

■  Sensitization  to  several  dispositional  aspects 

■  Both,  or 

■  Neither 

■  Teams  execute  2  TDGs 


Tactical  Decision  Games* 
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You  are  the  commanding  officer  of 
Company  G,  Battalion  Landing 
Team  2/2,  the  small  boat  company 
of  the  26th  Marine  Expeditionary 
Unit  (Special  Operations  Capable). 

■  Your  company  is  currently 
embarked  aboard  the  USS  Austin, 
and  it  is  part  of  a  combined  U.S.- 
Baklavarian  amphibious  task  force 
responding  to  an  escalation  of 
arms  smuggling  in  the  Adriatic 
Sea. 

■  Arms  smugglers  continue  to  use 
the  small,  uninhabited  islands 
along  the  central  Baklavarian  coast 
as  transshipment  points  for 
weapons  to  insurgent  groups 
operating  in  the  southern  Astorian 
Sea. 

■  Etc... 

^Marine  Corps  Gazette 


Analyses 
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■  Measures 

■  Counts  of  skills  observed  in  dialogues 

■  Self-reported  use  of  skills 

■  Correctness  of  solutions 

■  Analyses 

■  Evaluate  impact  of  training  &  sensitization  on 
outcomes 

■  Estimate  unique  contributions  of  cognitive  and 
dispositional  factors  using  hierarchical 
regression 
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Conclusion 
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Collaborative  Critical  Thinking 


■  Objective 

■  Define 

■  Measure 

■  Train  and 

■  Support 

■  Collaborative  Critical 
Thinking  for  teams 

■  Multi-expert 

■  Distributed 

■  Ad  hoc 

■  High  stakes 
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Backup 


Ap^HL _  Theory  Development: 

Example  of  Collaborative  Critical  Thinking 

■  Mike:  JFACC  Rear  ■  Gavan:  JFACC  Forward 

(CONUS  Junior  Analyst)  Analyst  in  charge  of 
calls  Gavan  to  discuss  the  mission  planning 
current  situation.  They  are 
using  NetMeeting  to 
share  information. 
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Example  of  Collaborative  Critical  Thinking 

The  Situation:  A  long-range  mission  is  in  progress  to  attack  pre¬ 
targeted  areas  in  Ichtar  and  West  Ichtar  and  is  scheduled  to 
commence  at  0800.  One  of  the  specific  targets  which  impacts  the 
entire  mission  is  a  fortified  SAM  site.  Electronic  Intelligence  has 
reported  that  the  site  has  remained  stationary  for  over  a  year.  In 
addition,  Imagery  out  of  Langley  reports  the  absence  of  any  support 
vehicles  necessary  to  facilitate  relocation.  Four  hours  before  the 
attack  begins.  Communications  Intelligence  out  of  NSA  reports  that 
the  fortified  SAM  site  is  indeed  on  the  move. 

A  group  of  JFACC  analysts  is  required  to  make  sense  of  all  this 

information.  The  location  of  the  SAM  site  influences  both  allocation 
of  friendly  resources  and  the  protection  of  friendly  forces.  One  of 
them  is  concerned  that  the  location  of  the  SAM  has  become 
uncertain. 

Mike  monitors  for  uncertainty  concerning  the  situation  and  plan. 


Theory  Development: 
Example  of  Collaborative  Critical  Thinking 
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■  Mike:  Gavan  we  need 
to  redirect  our 
friendlies  to  account 
for  SAM  A34's 
relocation. 


■  Gavan:  If  there's  a 
new  threat,  yes  we 
do.  Which  SAMs, 
briefly? 

Mike  &  Gavan  assess  the 
importance  of  resolving  this 
uncertainty 
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Example  of  Collaborative  Critical  Thinking 

■  Mike:  Blue  arrow, 
due  north  of  the  river 
Ichtar. 


Unmapped 
T unoro 


1CHTAR 


AVONSDALE 


ROMSDAL 


Wtdewall  Bay 


C0H0LEA5 


Theory  Development: 
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■  Gavan:  Ok.  I  thought 
this  SAM  was  fortified, 
stationary? 


Engineering 


Theory  Development: 
Example  of  Collaborative  Critical  Thinking 


■  Mike:  Negative. 
COMINT  has  just 
reported  that  the  SAM 
is  moving.  Here  is  a 
copy  of  that  report. 
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UUUUFM:  NSA/CSSTO:  JFACCSUBJ:  POSSIBLE  SAM  MOVEMENT(U) 
THE  SA-6  LOCATED  AT  32U345098  IS  POSSIBLY  PREPARING  TO 
MOVE  LOCATIONS.  VOICE  COMMUNICATIONS  BETWEEN 
OPERATORS  INDICATE  THAT  PREPARATIONS  NECESSARY  FOR 
THE  MOVEMENT  MAY  HAVE  BEGUN  AS  EARLY  AT  0030Z. 
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■  Gavan:  I  see  it. 
Didn't  ELINT  and 
IMINT  report  no 
movement  and  no 
support. 


Aptel _  Theory  Development: 

Example  of  Collaborative  Critical  Thinking 

■  Mike:  Roger.  ■  Gavan:  That  doesn't 

make  sense.  Doesn't 
COMINT  get  their 
information  from  the 
other  two? 


Gavan  identifies  a  source  of 


Centered  Engineering 
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■  Mike:  That's  my 
understanding,  but  I  will 
confirm  that. 


Gavan:  So,  we  should  check 
back  to  make  certain  these 
reports  are  correct.  Why 
don't  you  check  back  with 
IMINT  and  I'll  check  back 
with  ELINT  to  verify  this 
information.  We  still  have  a 
bit  of  time.  Ask  them  how 
conclusive  their  information 
is.  How  did  they  decide  this 
SAM  would  not  movdjf? 

Gavan  produces  a  plan  ti  >  refine 
their  confidence  in  the 
information. 
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Mike:  Shouldn't  we 
decide  on  a  time  to 
abort  the  mission  or 
at  least  to  make  a 
final  call? 


Gavan:  Yes.  Probably  the 
safest  thing  to  do  would 
be  to  cancel  the  mission  if 
we  aren't  certain.  That 
way,  no  friendlies  will  be 
compromised  due  to  a 
lost  SAM.  Let's  huddle  no 
later  than  0500  and  make 
a  final  call  no  later  than 


0600. 

Gavan  &  Mike  will  oipdt 
contingency  plan 
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■  Mike:  But  wouldn't  we 
miss  the  opportunity  to 
hit  these  other  sites?  Do 
we  know  why  we  are 
hitting  these  sites  today? 

Mike  monitors  for  sources  of 
uncertainty  and  risk \  and 
prompts  Gavan  to  help 
identify  them. 


